Does an Efficient Calibrated Forecasting Strategy Exist?

We recall two previously-proposed notions of asymptotic calibration for a forecaster making a sequence of probability predictions. We note that the existence of efficient algorithms for calibrated forecasting holds only in the case of binary outcomes. We pose the question: do there exist such efficient algorithms for the general (non-binary) case? Review of Calibrated Forecasting Glenn Brier, writing in the journal Monthly Weather Review in 1950, observed a challenge in assessing sequential probability forecasts [2]: Verification of weather forecasts has been a controversial subject for more than a half century. There are a number of reasons why this problem has been so perplexing to meteorologists and others but one of the most important difficulties seems to be in reaching an agreement on the specification of a scale of goodness for weather forecasts. In response, he proposed an objective scoring function which, if implemented by the forecaster, would lead to “calibrated” predictions. Yet a question which presumably did not occur to Brier is whether the latter is even computationally feasible. This question will be topic of the present note. Precisely, we would like to know whether there exists an efficient algorithm for forecasting sequences of outcomes that is asymptotically calibrated. As the forecaster observes the sequence y1, y2, . . ., from some given set K, she outputs a sequence of (potentially randomized) probability predictions p1,p2, . . . ∈ ∆(K), where pt may depend only on the past y1, y2, . . . , yt−1. Roughly speaking, a forecaster is asymptotically calibrated if her probability predictions match the outcome frequencies. Let τT (p) := {t : pt ≈ p}, and let δy ∈ ∆(K) be a point mass distribution on y, then our forecaster is calibrated if, for all p ∈ ∆(K), ∑ t∈τT (p) δyt |τT (p)| → p as T →∞. In the above definition we have been intentionally vague about the meaning of pt ≈ p, as well as our notion of convergence in the limit. To be more precise we shall distinguish between two notions of calibration: strong and weak. Definition 1 A function g : ∆(K) → R is called lipschitz if there exists some c > 0 for which |g(p)− g(q)| ≤ c‖p−q‖1 for every p,q ∈ ∆(K). We shall call a forecaster weakly calibrated if for every lipschitz “test function” g and any sequence y1, y2, . . . ∈ K we have CST (g) := calibration score w.r.t. g { }} { ∥∥∥∥ 1 T T ∑ t=1 g(pt)(pt − δyt) ∥∥∥∥ 1 → 0 almost surely as T →∞. (1) Let Ip, : ∆(K)→ R be the indicator defined as Ip, (q) := 1 when ‖p− q‖1 ≤ and Ip, (q) := 0 otherwise. A forecaster is called strongly calibrated if equation (1) holds for the indicator test functions g(q) = Ip, (q) for every p ∈ ∆(K) and > 0. Given a fixed , a forecaster is -strongly calibrated if, for every p ∈ ∆(K), the calibration score CST (Ip, ) ≤ for large enough T . It should be clear from the definitions that strong calibration implies weak, but it may not be as obvious that the reverse does not hold. It has been shown that for the binary case (K = {0, 1}) there exists a deterministic weakly calibrated forecaster [6, 7], yet any strongly calibrated forecaster must be randomized. The latter is demonstrated by a well-known counter example of Dawid [4]. (For a complete survey on strongly calibrated forecasting, see Cesa-Bianchi and Lugosi [3].) The (strong or weak) calibration property leaves much to be desired as a measure of “performance” of a forecaster. For one, a forecaster can be calibrated yet completely inaccurate even on “easy” inputs: If the outcomes simply alternates as (0, 1, 0, 1, 0, 1, . . .) then the trivial forecaster predicting 0.5 achieves asymptotic calibration. Yet calibrated forecasting remains a useful tool in many circumstances, particularly given that it is robust to arbitrary and potentially adversarial inputs. It can be shown, for instance, that no-regret learning algorithms can be constructed from strongly-calibrated forecasters; the same trick can be applied towards a strategy for Blackwell’s Approachability problem. Open Problem: Can We Calibrate Efficiently? The existence of a calibrated forecaster has been established for some time (strong [5] and weak [6]) in most cases via construction. Unfortunately, these constructions have typically been characterized by very greedy methods that give rise to inefficient algorithms. The most common approach is to take the probability space ∆(K) and cover it with an -grid and, for each grid point p, the algorithm maintains some statistic. To achieve -strongly-calibrated forecaster, the algorithm will potentially have to process the entire grid for each prediction pt. This story may have a happy ending. Recent results suggest that calibration may be achieved via efficient methods. Mannor et al. [7] developed a weakly calibrated forecasting algorithm that requires constant time and space for each prediction. In the present year’s COLT proceedings, Abernethy et al. [1] developed an -strongly calibrated forecasting algorithm which requires O(log 1 ) time yet O( 1 ) space per prediction. What’s the catch? Each of the above algorithm are applicable only to binary forecasting and do not extend to more general K. Mannor and Stoltz [8] have recently introduced work addressing the case where |K| > 2, utilizing Blackwell approachability, to obtain -strong calibration with time and space complexity that behaves like O(1/ |K|). As described in the table below, very little progress has been made towards efficient algorithms that achieve calibration in general. Strong Weak Binary (|K| = 2) O(log 1 ) time, O( 1 ) space [1] O(1) time/space [7] Finite-alphabet (|K| = n > 2) O(1/ |K|) time/space [8] ? (no improvement over [8]) We use the term “efficient” to denote an algorithm whose per-step complexity and memory requirements are poly in |K| and poly-logarithmic in (1/ ). Our concrete questions are: 1. Is there an efficient time and memory algorithm for -strong calibration for the binary case? 2. Is there an efficient time and memory algorithm for weak calibration for the general case? 3. Is there an efficient time and memory algorithm for strong calibration for the general case? Our best guess is that such an algorithm likely exists, for both the weak and strong case, in particular because we have not placed any restrictions on the rate at which the algorithm must achieve the calibration objective. On the other hand, the previously-discovered tricks which lead to efficient calibrated forecasters may be very special to the binary case and we would not be surprised if no such efficient algorithms exist when |K| > 2.